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State Policy Differences Greatly Impact AYP Numbers 
A Background Paper from the Center on Education Policy 

Introduction and Main Conclusions 

When Congress reauthorizes the Elementary and Secondary Education Act of 1965 (ESEA), 
much of the debate will undoubtedly focus on the accountability requirements added to Title I by 
the No Child Left Behind Act of 2001 (NCLB). Title I authorizes federal aid to school districts to 
educate low-achieving children in low-income areas. Among the most controversial of these 
NCLB requirements are the provisions for determining whether schools and districts have made 
adequate yearly progress (AYP) in raising student achievement in reading and mathematics. 

This background paper from the Center on Education Policy (CEP) explores some of the factors 
that have influenced recent trends in the national percentage of public schools that have not made 
AYP, out of the total number of schools that reported AYP results. This paper is intended to 
serve as a companion to the report, Update with 2009-10 Data and Five-Year Trends: How Many 
Schools Have Not Made Adequate Yearly Progress? (CEP, 2011). As discussed in more detail in 
that five-year trend report, the national percentage of public schools failing to make AYP rose 
from 29% in 2006 to an estimated 38% in 2010 and actually decreased in two of the interim 
years. 1 Although 38% is a record high percentage of schools not making AYP, it is still lower 
than what many observers had predicted by this point in NCLB implementation. 



1 AYP determinations are typically reported in the summer of a given year and are based on the results of tests 
administered in the school year that ended in the spring of that year. For example, AYP determinations for 2006 are 
based on test results from school year 2005-06. 



Our analysis focused on 10 large or medium-sized states that had the greatest increases or 
decreases in the number of schools not making AYP or had other noteworthy AYP trends: 
California, Florida, Illinois, Missouri, New York, North Carolina, Oklahoma, South Carolina, 
Texas, and Washington. The analysis revealed several factors, in addition to changes in student 
learning, that appear to account for some of the fluctuations in the national percentage of schools 
not making AYP and may help explain why these percentages have not escalated as quickly as 
some analysts have predicted. Our main conclusions are as follows: 

• National trends in the percentage of schools not making AYP have been affected 
disproportionately by trends in a limited subset of states. While the nation saw a modest 
overall increase between 2006 and 2010 in the percentage of schools failing to make 
AYP, trends were much more pronounced in some states. Steady growth in the number 
and percentage of schools not making AYP in four large or medium-sized statesS 
California, Illinois, Missouri, and FloridaS can account for a major portion of the overall 
national increase during the past five years. Dramatic jumps in the number of schools not 
making AYP in two statesS New York and Oklahoma 5 were a major contributor to the 
one-year increase in the national percentage from 2009 to 2010. By the same token, the 
one-year decrease in the national percentage from 2008 to 2009 was influenced to a 
notable extent by dramatic drops in the number of schools not making AYP in North 
Carolina and South Carolina and a significant decline in the large state of Texas. 

• Changes in state testing policies have slowed, or even reversed, increases in the 
number of schools failing to make AYP in several states. These changes include 
lowering cut scores for proficient performance on state tests, adopting new tests, and 
revising test administration policies, such as whether to count proficient scores from 
student nretestsOin AYP determinations. Without these fbrakes,Othe number of schools 
not making AYP would have grown at a faster rate, both for individual states and for the 
nation as a whole. 

• The NCLB “safe harbor” provision has also helped somewhat in keeping down the 
share of schools failing to make AYP in certain states. Very few states provide data on 
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the number of schools that made AYP solely as a result of the safe harbor provision, 
which gives credit for certain decreases in the percentage of students scoring below the 
proficient level. In Illinois, an example of a state that does report these data, the 
proportion of schools making AYP via safe harbor increased from 2% in 2006 to 15% in 
2010. While the percentage of Illinois schools not making AYP still went up overall 
during this period, it would have gone even higher without safe harbor. 

• The use of growth models appears to have had a limited impact on A YP trends in most 
of the growth model states analyzed. In Florida, Missouri, and North Carolina, the use of 
growth models, which track yearly gains in achievement for individual students and give 
schools credit for student improvement over time, has resulted in relatively small 
decreases in the number of schools failing to make AYP. Texas was an exception; in this 
state the application of a growth model appears to have been a key factor in a decline in 
the number of schools not making AYP. 

• In most of the states analyzed, the number of schools failing to make A YP increased 
substantially in the years when the state’s achievement targets went up. Even in states 
like Florida, Illinois, and Missouri that call for gradual annual increases in their 
achievement targets, student achievement gains have generally not kept pace with rising 
targets, resulting in greater numbers of schools not making AYP. In Oklahoma, which 
calls for somewhat steeper increases in achievement targets every three years, the number 
of schools not making AYP grew rapidly in 2010, a year when the targets rose. This 
finding has particular implications for states in which the targets are scheduled to go up 
sharply in the years approaching 2013-14; these states are likely to see big increases in 
the next few years in the number of schools not making AYP. 

• States that introduced new tests saw substantial short-term increases or decreases in 
the number of schools failing to make AYP. North Caro Una implemented new reading 
tests in school year - 2007-08, a decision that appears to have contributed to a substantial 
increase in the number of schools failing to make AYP that year. In Washington State, a 



3 




decrease in schools failing to make AYP in 2010 was associated with the introduction of 
new tests that year in grades 3-8 and high school. 

• In some states, changes in the number of schools not making AYP are largely 
attributable to changes in the cut scores defining “proficient” performance on state 
tests. In South Carolina, the decision to lower cut scores appears to have been a major 
reason behind a dramatic decrease between 2008 and 2010 in the number of schools 
failing to make AYP. Although several states have lowered their cut scores to deal with 
the demands of NCLB accountability, at least one state, New York, recently raised its cut 
scores, resulting in a large increase in the number of schools failing to make AYP. 

• North Carolina ’ s experience suggests that counting proficient scores from retests can 
reduce the number of schools failing to make AYP — especially in the short term. In 
school year 2008-09, North Carolina began counting proficient scores from retests of 
students who scored below the proficient level on their first attempt. The inclusion of 
these retest scores in AYP determinations apparently contributed to a significant increase 
in the number of schools making AYP via safe harbor in 2009 and in turn helped reduce 
the number of schools not making AYP. The safe harbor effect is likely to be short-lived 
because safe harbor determinations in future years will be based on comparisons with 
past years that include scores from retests. 

• Even if most or all states adopt common standards and common assessments, 
variations in state accountability policies could continue to make it impossible to arrive 
at meaningful comparisons about the performance of different states. Differences 
among states in academic standards, testing systems, accountability policies, and other 
areas make it impossible to compare AYP results in a meaningful way, either across 
states or within the same state over multiple years. While greater uniformity will come 
from the common core state standards being promoted by the nations governors and 
state superintendents and from the common assessments being developed by two national 
consortia, these actions alone will not produce a consistent state approach to 
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accountability and AYP. This issue should be recognized and possibly addressed in the 
ESEA reauthorization process. 

Background on AYP Requirements and Methods of Analysis 

The No Child Left Behind Act required states to set standards for academic proficiency in 
reading/English language arts and mathematics, and to develop tests for measuring studentsO 
progress toward these standards. School districts that receive Title I funds must administer these 
state tests annually to students in grades 3-8 and in one high school grade chosen by the state 
(usually grade 10 or 11). 

In addition, states had to set yearly targets, or annual measurable objectives (AMOs), for the 
percentages of students that must score proficient on state reading and math tests and for 
performance on at least one other academic indicator. For high school, this additional indicator 
had to include the graduation rate; for elementary and middle schools, states often chose the 
attendance rate. Under NCLB, these AMOs must rise periodically on a trajectory that leads to the 
ultimate goal of 100% of students reaching proficiency by the end of school year 2013-14. Some 
states have adopted an incremental approach that assumes relatively steady increases on the way 
to 100%, while others have nbackloadedO their trajectories by expecting smaller achievement 
gains in the earlier years, followed by much steeper gains in the years approaching 2014.' 

The AMOs play a critical role in the NCLB accountability system because they are used to 
determine whether schools and districts have made adequate yearly progress. Not only must the 
overall student population in a school or district meet every AMO, but each major racial, ethnic, 
and demographic student group in the school or district must also meet these targets. The 
subgroups whose performance is tracked for NCLB accountability include African American, 
Asian American, Latino, white, and (in some states) Native American students, and also include 
low-income students, English language learners, and students with disabilities. For example, a 



2 For more information about state AMOs and trajectories, see the 2008 CEP report. Many States Have Taken a 
"Backloaded” Approach to No Child Left Behind Goal of All Students Scoring “ Proficient . ” 
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school typically would not make AYP if the percentage of Latino students scoring proficient fell 
short of the state AMO of 80%. 

There are some exceptions to these basic requirements for determining AYP. One is a provision 
in federal guidance that allows states to exempt schools or districts from accountability for the 
performance of a particular subgroup if the number of test-takers in that subgroup is below a 
state-set minimum subgroup size. Another key exception is the lYafc harborO provision. A 
subgroup that falls short of the AMOs for proficiency in reading and math can still make AYP 
through safe harbor if the percentage of students scoring below proficient decreases by 10% or 
more, and if the subgroup meets the stated other academic indicators and the federal requirement 
to test 95% of the students in each subgroup. 

AYP determinations play a central role in the NCLB accountability system. Schools and districts 
receiving Title I funds that fail to make AYP for two consecutive years or more are identified for 
improvement and are subject to a series of consequences that become more stringent over time 
and culminate in a school having to undergo nrestructuringOof its governance and operations. 
Some states also apply these consequences to schools that do not receive Title I. 

Since the early years of NCLB implementation, various analysts have predicted that the number 
of schools not making AYP would increase rapidly in future years and would eventually include 
a majority of the nationd schools (e.g., Olson, 2002; Olson, 2005; Wiley, Mathis & Garcia, 2005; 
University of California-Riverside, 2008). To date, the direst projections have not come to pass. 
Over the past five years, the national percentage of public schools 3 not making AYP has changed 
relatively slowly, rising from 29% in 2006 to an estimated 38% in 2010. 4 Moreover, this 
percentage actually decreased slightly compared with the previous year in 2007 and 2009. 



3 The baseline number of schools used to calculate these percentages was the total number of schools that reported 
AYP results. This is a somewhat smaller number than the total number of schools in the nation or a particular state 
because states may exempt certain schools from AYP determinations, such as new schools that have existed for a 
year or less, schools that do not serve any of the grades tested by NCLB, short-term schools that are not attended by 
any students for more than a year, and schools with unusual circumstances. 

4 A11 of the figures for school year 2009-10 in this paper are based on preliminary estimates of numbers of schools 
not making AYP. The status of schools could change in the coming months due to school district appeals of state 
AYP determinations, waivers, and other factors. Official AYP numbers for 2010 based on State Consolidated 
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Why has the national percentage of schools not making AYP fluctuated over the past five years, 
and why does it remain under 40% despite predictions and rising AMOs? As a starting point for 
exploring these questions, a CEP consultant analyzed the state-by-state data collected for and 
presented in CEP& five-year AYP trend report (CEP, 2011). The data for 2006 through 2009 
came from the State Consolidated Performance Reports submitted to the Ei.S. Department of 
Education. The data for 2010 were estimates collected from what we believe are the most 
reliable sources available at the time of our research, including state Web sites and direct 
communications with state education agency personnel. 5 The analysis for this background paper 
focused on two time periods: 2006 through 2010, which is the entire span covered by the five- 
year trends report; and 2008 through 2010, which includes the three most recent years of AYP 
results and encompasses both a decline and a rise in the national percentage of schools not 
making AYP. 

Our initial review of the state-by-state AYP data revealed that national changes in the percentage 
of schools not making AYP are often disproportionately influenced by large decreases or 
increases in a relatively small subset of states. To investigate the reasons underlying these 
national changes, a CEP consultant analyzed factors affecting AYP trends in a manageable group 
of 10 large or medium-sized states. Nine of these statesS California, Florida, Illinois, Missouri, 
New York, North Carolina, Oklahoma, South Carolina, Texas, and WashingtonS had the 
greatest increases or decreases in the number of schools failing to make AYP during one or both 
of the time periods analyzed. The tenth state, South Carolina, experienced a steep drop in the 
number of schools not making AYP between 2008 and 2009, spurred mainly by the stated, 
decision to lower its cut scores defining proficient performance. The analysis did not include 
small states because they have relatively few schools, and even a large percentage fluctuation in 
their schools not making AYP would have a minor impact on the national total. Information 
sources for the analysis included state Web sites, state accountability plans, research studies, and 
other sources. 

Performance Reports are not yet available from the U.S. Department of Education. When they do become available 
later this year, CEP will release a report with final AYP figures for 2010. 

5 A full list of sources is included in the appendix to the five-year trend report (CEP, 2011), available at www.cep- 
dc.org. 
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These 10 states fall into four groups, based on shared patterns in the number of schools not 
making AYP: 

1. States with mostly steady increases between 2006 and 2010 (California, Illinois, 
Missouri, and Florida) 

2. States with very large increases in 2010, after years of low numbers or declining trends 
(New York and Oklahoma) 

3. A state with decreases over time (Texas) 

4. States with ups and downs (North Carolina, Washington, and South Carolina) 

The main sections of this paper examine factors, in addition to changes in student learning, that 
appear to have shaped AYP trends in these four groups of states. It is important to keep in mind 
that states vary considerably in their AMOs, testing systems, accountability policies, 
demographics, and other characteristics. For that reason, AYP results are not comparable across 
states, and a state with a high percentage of schools failing to make AYP should not be assumed 
to have a weak educational system. 

States with Mostly Steady Increases over Five Years 

Four statesS California, Illinois, Missouri, and FloridaS had a major impact on the overall 
national increase between 2006 and 2010 in the number and percentage of schools failing to 
make AYP. These four states are discussed below, in order of their impact on the national 
aggregate number of schools failing to make AYP. 

Two of these states, Florida and Missouri, have adopted growth models for determining AYP, 
but this seems to have had a relatively minor impact on the numbers of schools failing to make 
AYP. 

Rather, rising AMOs are the primary factor underlying the increases in schools not making AYP 
in these states. In three of these statesS Illinois, Missouri, and FloridaS the AMOs have risen 
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gradually each year in equal increments. In the fourth state, California, the AMOs increased just 
once every three years until school year 2007-08, then began to increase annually in equal 
increments. In none of these four states has student achievement gains kept pace with the annual 
AMO increases. 



California 



California 


2006 


2007 


2008 


2009 


2010 


Percentage of schools not making AYP 


34% 


33% 


48% 


50% 


61% 


Number of schools not making AYP 


3,274 


3,217 


4,690 


4,921 


6,061 



Both the number and percentage of California schools not making AYP have climbed 
continuously since 2008, a period when the state® AMOs increased yearly instead of every three 
years. Student achievement in many schools did not keep pace with rising AMOs, and the 
number of schools failing to make AYP grew by 1,371 schools between 2008 and 2010. Because 
California is a heavily populated state with several thousand schools, these increases had a 
significant impact on the national number and percentage of schools not making AYP. 



Illinois and the role of safe harbor 



Illinois 


2006 


2007 


2008 


2009 


2010 


Percentage of schools not making AYP 


18% 


24% 


32% 


41% 


51% 


Number of schools not making AYP 


812 


896 


1,200 


1,553 


1,999 



Illinois has seen a steady increase since 2006 in the number and percentage of schools failing to 
make AYP. This number grew by 1,187 schools between 2006 and 2010, and by 799 schools 
between 2008 and 2010. 



Illinois is one of the few states that regularly report the number and proportion of schools making 
AYP specifically due to safe harbor. The table below shows the number of Illinois schools that 
made AYP solely as a result of safe harbor during the past five years, as well as the percentage 
that these schools represent out of all Illinois schools that made AYP in a given year. As the 
table indicates, both the number and percentage of schools making AYP due to safe harbor have 
clearly increased over the past four years. Still, this percentage has never exceeded 15%, and the 
growth in safe harbor schools has not been sufficient to counteract the general increase in 
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schools not making AYP. Were it not for the growing reliance on safe harbor, a greater share of 
Illinois schools would be failing to make AYP. 



Number and percentage of Illinois schools that made AYP due to safe harbor 



Year 


Number of Schools 


Percentage of All Schools Making AYP 


2006 


65 


2% 


2007 


169 


6% 


2008 


182 


7% 


2009 


231 


10% 


2010 


277 


15% 



Table reads: The number of Illinois schools that made adequate yearly progress as a result of NCLB® safe harbor provision grew 
from 65 in 2006 to 277 in 2010, or from 2% of all Illinois schools that made AYP to 15%. 

Source: Center on Education Policy based on data collected from State Consolidated Performance reports and Illinois State Board 
of Education, 2010. 



Safe harbor has played a similar role in other states that were not the main focus of our analysis. 
In Colorado, for example, the percentage of schools making AYP solely due to safe harbor rose 
from 4% in 2006 to 13% in 2010, according to our calculations based on state data (Colorado 
Department of Education, 2010). It is noteworthy that Colorado uses two different approaches to 
determine whether schools qualify for safe harbor. In addition to the standard way, Colorado also 
uses a nmatched safe harbor 6 approach, whereby safe harbor is determined based only on scores 
from individual students who were tested in both of the years being compared. Several schools in 
Colorado that did not meet the standard safe harbor criteria did make AYP through matched safe 
harbor. 

Maryland is another state with an increase in the percentage of schools that made AYP solely as 
a result of safe harbor. This percentage rose from 14% in 2006 to 20% in 2010, according to our 
calculations based on state data (Maryland State Department of Education, 2010). It should be 
noted, however, that Maryland also made changes in assessment policies regarding retests and 
the testing of students with disabilities that might have affected both test score trends and safe 
harbor determinations. Still, it seems evident that without safe harbor, even more schools in 
Colorado and Maryland would have failed to make AYP. 
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Missouri 



Missouri 


2006 


2007 


2008 


2009 


2010 


Percentage of schools not making AYP 


29% 


46% 


57% 


63% 


64% 


Number of schools not making AYP 


603 


975 


1,261 


1,378 


1,407 



The percentage of Missouri schools not making AYP has increased steadily since 2006. The 
number of schools failing to make AYP grew by 804 schools between 2006 and 2010, including 
an increase of 146 schools between 2008 and 2010. 



In school year 2007-08, Missouri began using a growth model to make AYP determinations, but 
this does not appear to have resulted in a substantial decrease in the number of schools failing to 
make AYP. 



Florida 



Florida 


2006 


2007 


2008 


2009 


2010 


Percentage of schools not making AYP 


71% 


66% 


76% 


77% 


86% 


Number of schools not making AYP 


2,281 


2,141 


2,507 


2,564 


2,952 



In 2006, 71% of Florida schools failed to make AYP. This figure declined to 66% in 2007, and 
then increased in each succeeding year through 2010, when it stood at 86%. The number of 
schools failing to make AYP grew by 671 schools between 2006 and 2010, and by 445 schools 
between 2008 and 2010. 



Florida started using a growth model in school year 2006-07. According to a recent U.S. 
Department of Education study of the initial group of states that participated in the growth model 
pilot program, 5% of the public schools in Florida that made AYP in 2007 and 2008 did so via 
the growth model (Hoffer et al., 2011). Therefore, the initial implementation of the growth model 
partly explains the reduction in schools failing to make AYP in 2007 but has not been sufficient 
to counteract the general upward trend in schools failing to make AYP. 
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States with Very Large Increases in 2010 



New York and Oklahoma were major contributors to the national increase in schools not making 
between 2009 and 2010. Both states had tremendous increases in these schools in 2010. 



In both states, this jump in the number of schools failing to make AYP was associated with an 
increase in standards for student performance. Oklahoma^ AMOs went up substantially in 
school year 2009-10, while New York raised the cut scores students must reach on its state tests 
to be considered (proficient. 6 



New York 



New York 


2006 


2007 


2008 


2009 


2010 


Percentage of schools not making AYP 


29% 


20% 


16% 


12% 


38% 


Number of schools not making AYP 


1,264 


875 


737 


532 


1,670 



The percentage of schools failing to make AYP in New York started at the national average of 
29% in 2006, then fell to a low of 12% by 2009. The next year, in 2010, the percentage more 
than tripled to 38%. This leap in the number of New York schools failing to make AYP affected 
the national aggregate increase to a notable degree. 



New York does not use a growth model to calculate AYP. The stated, accountability plan 
provides for annual incremental increases in AMOs, and this no doubt contributed to an increase 
in schools not making AYP. But the state made another change in school year 2009-10 that had a 
greater impact on AYP determinationsd it substantially raised the cut scores for proficient 
performance on its English language arts and math tests for grades 3-8. The cut scores for the 
basic level of achievement were also increased. 



According to the New York State Department of Education (2010a), this change was intended 
nto align [proficiency standards] with college-ready performance. 6 After raising the bar for 
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proficiency, the number of students across grades 3-8 who met or exceeded the new cut scores 
dropped between 2009 and 2010 from 77% to 53% in English and from 86% in math to 61%. 6 



Oklahoma 



Oklahoma 


2006 


2007 


2008 


2009 


2010 


Percentage of schools not making AYP 


11% 


12% 


7% 


11% 


41% 


Number of schools not making AYP 


198 


220 


126 


190 


731 



The percentage of schools failing to make AYP in Oklahoma declined somewhat from 2006 to 
2008, rebounded in 2009, and then almost quadrupled in 2010. 



Oklahoma® AMOs increased once every three years through 2010. Therefore, the large increase 
in schools failing to make AYP in 2010 was associated with a rise in AMOs after three years of 
stable targets. It should be noted, however, that when the last increase in AMOs occurred in 
2006-07, there was no equivalent rise in schools failing to make AYP. 



Oklahoma has backloaded its trajectory toward 100% proficiency, so its AMOs are scheduled to 
go up in 201 1-12 and again in both 2012-13 and 2013-14 (CEP, 2008). If the 2010 figures are 
any indication, Oklahoma and other states with backloaded trajectories are likely to see big 
increases in the next few years in the number of schools not making AYP. 



Decreases over Time in Texas 



Texas 


2006 


2007 


2008 


2009 


2010 


Percentage of schools not making AYP 


19% 


9% j 


15% 


5% 1 


5% 


Number of schools not making AYP 


1,519 


664 


1,109 


353 


368 



About 19% of Texas schools did not make AYP in 2006, well below the national average. This 
share fell to 9% in 2007, grew somewhat to 15% in 2008, then dropped again to 5% in 2009 and 
stayed there in 2010. The number of schools not making AYP declined by 1,151 between 2006 



6 The New York State Department of Education requested a waiver from the U.S. Department of Education to allow 
AYP determinations for NCLB purposes to be based on the old cut scores for 2010. New York has also received 
approval from the U.S. Department of Education to adjust its AMOs for the 2010-11 through 2013-14 school years 
(New York State Department of Education, 2010b). 
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and 2010, and by 741 between 2008 and 2010. Thus, during both time periods, the trend in Texas 
moved strongly in the opposite direction from the national trend. The substantial reductions in 
the number of schools failing to make AYP in Texas, plus the decreases in North Carolina and 
South Carolina discussed later, help to explain why the national increase in schools failing to 
make AYP has not been greater. 

The Texas accountability plan provides for annual increases of equal increments in AMOs. The 
key factor in the declining number of Texas schools not making AYP appears to be the 
implementation of the stated; growth model, called the Texas Projection Measure, in 2008-09 
and beyond. According to the Texas Education Agency (2009), 528 schools made AYP in 2009 
that would not have done so without the use of this growth model. These schools accounted for 
70% of the decrease in the number of schools failing to make AYP in Texas between 2008 and 
2009. 



States with Ups and Downs 

North Carolina, South Carolina, and Washington State have experienced considerable ups and 
downs in recent years in the number and percentage of schools not making AYP. In all three 
states, the number and percentage of schools failing to make AYP in 2010 remained below the 
highest point reached in a previous year. 



North Carolina 



North Carolina 


2006 


2007 


2008 


2009 


2010 


Percentage of schools not making AYP 


56% 


55% 


69% 


29% 


42% 


Number of schools not making AYP 


1,310 


1,267 


1,659 


728 


[ 1,062 



The percentage of North Carolina of schools failing to make AYP fluctuated in the past five 
years in 20065 peaking at 69% in 2008, plunging the next year to 29%, and escalating again to 
42% in 2010. 



In recent years, North Carolina has made several changes in its testing program and its methods 
for calculating AYP. One of these changcsS the introduction of growth modelsS appears to have 
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had minimal impact on AYP trends, while the other changes seem to have influenced the number 
of schools not making AYP in complex and interrelated ways. 

In school year 2005-06, North Carolina began using a growth model for AYP determinations. 
According to the U.S. Department of Education growth model study, only 1% of the North 
Caro Una public schools that made AYP in 2007, and less than 1% in 2008, did so via the growth 
model (Hoffer et al, 2011). While comparable data are not available for later years, it seems very 
unlikely that the growth model had a significant impact on trends in the number of schools 
failing to make AYP in North Carolina. 

Beginning in 2008, North Carolina implemented new reading tests in grades 3-8, along with new 
cut scores for various achievement levels and a new baseline for its AMOs. These changes 
appear to have contributed to the substantial increase in the number of schools failing to make 
AYP that occurred in 2008. The AMOs for 2008 remained in effect through 2009-10. The state® 
AMOs are scheduled to increase substantially in 2010-11 and again in 2013-14. 

Beginning in school year 2008-09, North Carolina made several changes in the ways tests were 
administered and used for AYP determinations. Together, these changes most likely led to a 
substantial increase in the number of students deemed to be proficient. The broadest change was 
to allow students who missed the proficient mark on their first try to retake a test, and to count 
scores of proficient or above from this first retest for AYP purposes. This policy was introduced 
for students in grades 3-8 in 2008-09 and for high school students in 2009-10 (North Carolina 
Department of Public Instruction, 2009). 

In 2008-09, North Carolina also began counting, as part of its AYP calculations, proficient 
scores from students who had exited the subgroup of students with disabilities within the 
previous two years. In other words, the percentage proficient for the subgroup of students with 
disabilities could include proficient scores from students who had been identified for special 
education services in one or both of the preceding two years but who were no longer considered 
eligible. 
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Safe harbor was also a likely contributor to the shifts in the number of North Carolina schools 
not making AYP. Although North Carolina reports data on the number of schools that have 
qualified for safe harbor, it does not break out which of these schools made AYP solely because 
of safe harbor. North Carolina reported that 447 schools met the safe harbor criteria in 2007 ; this 
number declined to 350 in 2008, then almost tripled to 987 in 2009 before dropping back to 413 
in 2010. 

While some of these schools might also have made AYP through the stated, growth model or 
confidence intervals (a sort of statistical margin of error), there is reason to believe the initial use 
of retest scores contributed to a large, short-term increase in the number of schools making AYP 
solely due to safe harbor. School district data and news reports indicate that the combination of 
retest scores and safe harbor was a factor in the rising number of schools making AYP in some 
of the stated larger districts, including Wake County, which includes Raleigh, and Guilford 
County, which includes Greensboro (Wake County Public School System, 2011; Clark, 2010; 
Guilford Education Alliance, 2009 and 2010). In addition, the state board of education reported 
that counting retest results for the first time provided a one-time boost in the percentage of 
elementary and middle school students scoring at or above the proficient level in 2009, and in the 
proportion of high school students reaching proficiency in 2010 (North Carolina State Board of 
Education, 2010a). 

Since safe harbor requires a reduction in the number of students scoring below proficient 
compared with the previous year, the AYP benefits of the interaction between safe harbor and 
retests are likely to fade after the first year of implementing retests. This is because safe harbor 
determinations in future years will be based on comparisons with past years that include scores 
from retests. This may explain why the total number of schools making AYP in North Carolina 
dropped considerably between 20096 the first year of implementing retests in grades 3-86 and 
2010, even though the AMOs did not increase (North Carolina Board of Education, 2010b). 

But even without the safe harbor benefit, the use of retest scores is likely to continue to help 
some schools make AYP. 
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Washington State 



Washington 


2006 


2007 


2008 


2009 


2010 


Percentage of schools not making AYP 


16% 


35% 


62% 


58% 


46% 


Number of schools not making AYP 


341 


734 


1,307 


1,228 


968 



Washington State experienced huge jump in the percentage of schools that did not make AYP 
between 2007 and 2008 and a significant decline between 2009 and 2010. Washington does not 
use a growth model. Its accountability plan calls for increases in AMOs once every three years; 
the AMOs rose in 2008, remained constant for the next two years, and are rising again in 2010- 
11 . 



The large increase in schools not making AYP in 2008 was associated with a relatively large 
increase in AMOs. The decrease in 2010 was associated with the introduction of new tests that 
year in grades 3-8 and high school. 



South Carolina 



South Carolina 


2006 


2007 


2008 


2009 


2010 


Percentage of schools not making AYP 


62% 


63% 


80% 


50% 


48% 


Number of schools not making AYP 


668 


711 ] 


893 


728 


521 



In South Carolina, the number of schools failing to make AYP rose from 2006 through 2008, 
then dropped by 372 schools between 2008 and 20105 a steep decline from 80% not making 
AYP to 48%. The real drop occurred at the elementary and middle school level; the number of 
South Carolina high schools failing to make AYP actually increased between 2008 and 2009 and 
remained stable between 2009 and 2010. 



The main reason behind the decreases in elementary and middle schools not making AYP in 
2009 and 2010 appears to be the state legislature® decision to lower the cut score for 
fproficientO performance on the tests used for grades 3-8 (South Carolina Department of 
Education, 2010). The state declared a score of HbasicOon the state test to be equivalent to 
fproficientO for purposes of NCLB compliance; previously, the state had used the fproficientO 
score on the state test for this purpose. This is one of the more obvious and direct state actions to 
lower academic standards, with the effect of increasing the number of schools making AYP. 
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South CarolinaQ AMOs increased in school year 2007-08, and the percentage of schools failing 
to make AYP that year was much higher than in the two previous years. The AMOs remained 
constant through 2009-10 but are scheduled to rise dramatically in 2010-11 and 2013-14. 

Implications of State Policy Variations 

As the preceding discussion illustrates, states have different testing and accountability policies 
that affect their numbers of schools not making AYP. Substantial variations exist among states 
not only in the policies discussed in this paperd such as proficiency cut scores, use of growth 
models, and inclusion of scores from rctcsts6 but also in other testing and accountability 
policies, including academic content standards, test content and difficulty, minimum subgroup 
sizes, and use of confidence intervals, to name just a few. Even within the same state, these 
policies often change over time. It is increasingly clear that these variations among states make it 
impossible to meaningfully compare progress in raising student achievement, either across states 
or within the same state over multiple years. 

The specific AYP provisions of NCLB were adopted in part to create more consistency in 
accountability across states than had existed under the predecessor statute, the Improving 
AmericaQ Schools Act of 1994 (Goertz, Duffy & Le Floch, 2001). Over the years of NCLB 
implementation, however, many new sources of state variation have emerged. 

State variations are likely to continue to exist even if most or all states adopt the common core 
state standards for mathematics and English language arts being developed by state leaders in 
cooperation with the National Governors Association and the Council of Chief State School 
Officers. The core standards deal only with the academic content students should learn, leaving 
other critically important decisions about accountability to the states. While variations in state 
content standards are a substantial source of state variation currently, they are far from the only 
one, and not necessarily the most significant. 
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The common assessments aligned with the common core standards that are being developed by 
two consortia with many state members should result in more uniform assessment systems 
among the states participating in each consortium, especially if these states agree to common cut 
scores for basic, proficient, and advanced achievement. But even with common assessments and 
cut scores, it could still be difficult or impossible to arrive at meaningful comparisons of AYP 
trends across states if the options to make different decisions about accountability are still 
available to states. In addition, the two consortia are envisioning different approaches to 
assessment, and it is not clear to what extent the two assessment systems emerging from their 
work will be linked. 

Further, there is no clear indication that these types of state variations would be reduced under 
the Obama Administrations nBlueprintOfor the reauthorization of ESEA (U.S. Department of 
Education, 2010). The Blueprint provides very few details about which accountability provisions 
the Administration would substitute for current AYP policies, and there is no indication that the 
new provisions would be more consistent across states and time periods than the current AYP 
criteria. The Blueprint encourages states to adopt the common core standards, but also 
emphasizes flexibility for states in identifying schools that are not performing adequately and 
designing improvement strategies for such schools. 

In other words, even if academic standards, assessments, and cut scores are more uniform across 
many states, it still may not be possible to meaningfully compare the performance of these states 
if they continue to have considerable flexibility about the accountability part of the equation. 
Given the flexibility in some of the proposals before Congress, state variations in accountability 
systems could be even more diverse. While such variation may be virtually inevitable in a 
federalist system of educational finance and governance, at the least this issue should be 
recognized, and possibly addressed, by Congress in the ESEA reauthorization process. 
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